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SUMMARY 

Hie project, "Improving Research Capabilities o£ Large City School 
Systems was conducted by The Council of the Great City Schools as one 
of a series of activities designed to strengthen the research and evalua- 
tion divisions within its member districts. 

The project was designed as a set of four training sessions. Two of 
these were for research directors from the 21 school districts comprising 
the Council, and two were for members of research and evaluation staffs. 
Generally, the sessions for research directors were intended to increase 
knowledge in broad and rapidly developing areas within the research and 
evaluation field; the purpose, in other words, was not to develop specific 
skills but to provide the basis for longer-range decisions aimed at improv- 
ing research and evaluation services. The sessions for staff members, on 
the other hand, were narrow and intensive and were designed for the devel- 
opment of specific skills, , 

Che research directors session was held in Washington, D.G. , in Nov- 
ember of 1969 and dealt with infomiation systems and with evaluation of 
multiple project programs. The second, research director's session, held 
in Vail, Colorado, in May, 1970, was concerned with systems approaches to 
the solutions of educational problems. 

The first session for research and. evaluation staff was in Memphis > 
Tennessee 9 in February , 1970 * arid was concerned with measurement problems 
in the affective domain. In the second staff session, held in Monmoiath, 
Oregon , during September of 1970, a different approach was' followed. Under 
contract, the Teaching Research Division of the Oregon State System of 
Higher Education offered individualized instruction in four areas : proposal 
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writing, instructional systems, measurement, and evaluation. 

All indications are that the program was highly successful and that 
research and evaluation capabilities in large city school systems were 
indeed improved through it. 
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INTRODUCTION 



The research arid evaluation capabilities of large city public school 
systems have taken on a greatly increased importance in recent years. On 
a local level, school systems need valid information for improvement of 
educational programs and for facilitation of decision making , On a na- 
tional level, the effects of federal expenditures on strengthening achieve- 
ment of youth require systematic evaluation to justify allocation of re- 
sources, The possibility of meeting both local and national needs for 
valid information cannot be attained unless steps are taken to strengthen 
the operational capabilities of research and evaluation divisions in large 
city school systems. 

As recently as five years ago, the major responsibilities of Bureaus 
of Educational Research in public school systems were characterized by 
testing programs and administrative data gathering. Increasingly, however , 
the direction of research and development activities has turned. The mag- 
nitude of social problems and criticism of public schools has resulted in 
a redirection of efforts to include greater involvement in evaluation and 
experimental research related to current problems facing education. Ad- 
ditional impetus has been provided by ESEA Title I requirements for system- 
atic evaluation of projects. The consequence of these events is tnat public 
school research organizations have needed more adequately trained personnel 
to design, implement, and disseminate useful evaluative studies. 

The competence of public schoof research organizations has implica- 
tions beyond the operational effectiveness of any single school system. 
National consequences have been born. The efforts to improve education for 
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culturally disadvantaged children are directly related today to the compet 
ence of school research personnel to assess strengths and weaknesses of 
programs initiated with ESBA funds . Valid feedback of information is a 
critical element for local program development. Equally important, such 
feedback can enable state and federal agencies to assess the impact of 
dollar expenditure on the improvement of public education. 

The Council of the Great City Schools, in initiating and implementing 
the project "Improving Research Capabilities of Large City School Systems, 
completed the first stage of a systematic program to upgrade research comp- 
etence in twenty-one of the largest city school systems in the country . 

Directors of Research were involved in two three -day training seminars. 
The first of these dealt with evaluation as an information system for pro- 
ject development in addition to evaluation of multiple project programs. 

The second of the seminars examined systems analysis as, an approach to 
problem solving in education. 

Staff members within research departments of the Great Cities school 
systems participated in two concentrated training workshops. One of these 
sessions was directed at measurement in the affective domain. The other 
session consisted of individualized instruction in four areas. 

1) Proposal writing, 2) Instructional systems , 3) Measurement, and 

4) Evaluation. 

In addition, there were two meetings of the Superintendents, Assistant 
Superintendents, and board members of the twenty -one participating large 
city school systems. At both gatherings , emphasis was placed on research 
and. evaluation as a means to improve decision making. 

These sessions will be described in greater detail in the sections 

which follow. 

ERIC : ; . • v r 
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OBJECTIVES 

The following objectives guided program planning, implementation 
and evaluation: 

a. To improve the quality of program evaluation in 
large city public school systems. 

b. To develop a means for continually upgrading public 
school research personnel in gathering, analyzing, 
and reporting evaluative data. 

c. To improve communications between research personnel 
in public school systems, universities, and the U.S. 

Office of Education in relation to evaluation problems. 
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PROGRAM PROCEDURES 



"Improving Research. Capabilities of Large City 1 School Systems" cair d 
for implementation of a research training program in which the Council of 
the Great City Schools, in consultation with Office of Education Personnel 
and individual consultants, engaged in training activities leading to 
increased competence on the part of public school research personnel. This 
was achieved by updating and expanding the knowledge and capabilities of 
experienced personnel presently employed in the participating districts. 

Council staff, its Research Steering Conmittee, consisting of research 
directors from member cities, and outside consultants, worked together in 
planning for and conducting the actual training program. The training acti- 
vities themselves consisted of seminars, workshops, and individualized in- 



struction. 

There were four operational elements to the project: 

1) Orientation sessions for Superintendents, Assistant Super- 
intendents, and school board members, 

2) Intensive seminars for research directors of the partici- 
pating districts, 

3) Workshops for research and evaluation staff members. 

4) Organization of cooperative inter-school system efforts 

related to the study of common problems facing urban R 
school systems . ' . ' ' : •_ \ 

Orientation sessions were designed to provide top level administrators 
and school board members with tlie most current information and thought con- 
ceming the relationship of research and evaluation to operations of school 
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Research Director 1 s Sessions . - -Seminars tor research directors pro- 
vided for presentations and discussions of new developments related to 
data collection and analysis as well as approaches to evaluation in fiwld 



settings . 

In November, 1969, the twenty-one urban directors of research met in 
Washington, D. C, (See Appendix B) , The topics of discussion were 
’’Evaluation As an Information System for Project Development and 
"Evaluation of Multiple Project Programs”. From the U.3. Office of 



Education, Dr. Karl Hereford, Director of Program Planning of BESE, 
explained the issues related to evaluation of multiple programs. 

Dr. Francis A.J. Ianni of Columbia University, Dr. Richard Jaeger of 



Stanford University, and Dr. Robert Stake of the University of Illinois 
provided input from the higher level educational institutions. Finally, 
the Council research directors themselves, as well as representatives 



from urban school systems which were not members of the council, con- 
tributed practical operational models of evaluation. At the same 
conference site, the superintendents and board members of the partici- 
pating twenty-one members of the Council were gathered to discuss the 
goals and priorities of the Council with respect to research. Dr. 
James Gallagher of the U.S. .Office of Education added governmental 



information to the program. 

In May, 1970 , the research directors of the participating cities 
met in Vail, Colorado to discuss "Systems Viewpoint in Project 
Development". Dr. Desmond Cook and his associates, Dr, John Skalski 
and Dr. Gregory Trzebiatowski from Ohio State University 








to present not only basic systems concepts but also the tools and 
techniques used by a systems manager. Mil ti -project management was 
discussed in addition to single project development and evaluation. 

From a Council member city, Dallas, Texas, Dr. Rogers Barton provided 
input on performance contracting. Additional involvement of the research 
directors themselves occurred in several panel discussions on such 
topics as; "Implications of Using the Systems Approach for the Research 
Director" and "What Does Systems Analysis Mean Relative to Program 
Development and Evaluation" and "Where Do We Go From Here in the School 
Research Operation?" Indicative of the active interest in the topic of 
this seminar was the fact that many cities at their own expense sent more 
than one participant to the meeting. 

Staff Member 1 s Sessions . --Research and evaluation staff members 
attending the training program from the participating school districts 
reviewed statistical techniques . evaluation models , au^ data collection 
instruments . The participating individuals were experienced professionals 
who possessed knowledge and skill related to research. The purpose of 
the training program was to upgrade and update these capabilities. 

The first workshop for research and evaluation staff was held in 
February, 1970, in Memphis, Tennessee. In a meeting prior to the work- 
shop the Steering Committee of the research directors had decided that 
the topic of "Measurement in the Affective Domain" would be valuable to 
members of their staff. Dr. David Orr laid the groundwork for discussion 
by giving a historical background and speaking to the theoretical issues' 
in measuring affect. Follow up to this general introduction was provided 
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by Dr. Marvin Shaw of the University of Florida, Gainesville. Dr. Shaw- 
considered the general nature of attitude formation and change and also 
gave an overview of available measuring instnments. Particular 
emphasis was given to various problems identified by the participants 
themselves and on areas of concern in public education. Dr. Everett 
Rogers of Michigan State University presented specific attitude measuring 
methods focusing mainly on the Likert techni .e and on sociometric 
analysis. Dr. Bradley Greenberg, also of Michigan State University, 
addressed the group on the semantic differential and led the members 
in development of their own scales. After each topic of discussion 
the large body of participants broke into small groups, with an expert 
in each group, for individualized instruction and informal attention to 
specific problems. In addition, in an evening program presentations 
were given by the research staff on individual experiences within their 
particular city in developing and using affective instruments. Hie 
informal small groups combined with individual presentations created a 
good rapport and friendly spirit among the participants and between the 
participants and consultants. 

Hie second workshop for the staff of the research directors was 
held in Monmouth, Oregon, at the Oregon College of Education, in 
September, 1970. This session was planned and handled in different 
manner than the others. ^ x 

Dr. Jack Idling, of the Teaching -Research Division of the' Oregon 
State System of Higher Education, met with the Council's Research 
Steering Committee, presented training materials already developed by 
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Teaching Research, and suggested an individualized training approach 
which would allow each research director some latitude in determining 
which shills he would like his staff members to develop. The Steering 
Committee voted to follow Dr. Edling’s suggestion. 

At the training session, research and evaluation staff members from 
Council cities began in one of areas: proposal writing, instructional 

systems, measurement, and evaluation. Tests were given to determine entry 
level skills, and instruction was highly individualized by the staff of 
Teaching Research, working under the direction of Dr. Dale Hamerus and 
Dr. Janes Blaird. Participants moved at their own rates, and if they 
reached proficiency level in their initial area, they were allowed to 
choose a second area and begin work there. 

Some 30 research and evaluation staff members, 21 paid for by the 
project and the remainder sent at the district’s expense, attended the 
three -day session. Proficiency tests given by Teaching Research con- 
firmed that intended skills were developed and letters from participants 
to the Council indicated their satisfaction with the program. 

As mentioned previously, the Superintendents from the twenty-one 
member cities of the Council participated in the conference in 
Washington, D. C. , in November, 1969, In addition, a meeting was 
held among the Superintendents in May, 1970, in Buffalo, New York. 

Here, too, the Proj ect Director^ presented information regarding the 
programs which had already taken place, in an effort not only to keep 



the superintendents .. .informed on the content of the programs but also 

5 research departments and the 




decision makers . Suggestions for future programs were also introduced 



J discussed 
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REaJLTS 

It is difficult to measure directly the fulfillment of the original 
objectives of the project "Improving Research Capabilities of Large City 
School Systems." The activities were planned to insure the introduction 
of new ideas, methods, and techniques into the great city schools and to 
upgrade their research and administrative capabilities. Further , the de- 
sign of the project created a unique learning experience by bringing to- 
gether field authorities and practicing school personnel in a close work 



ing, as opposed to lecture, setting. 

The inservice training sessions were structured to meet the needs 



of participants, and the sessions also attend ted to consider the practi- 
cal relevant topics. Before each workshop , a Steering Committee repre- 
senting twenty -one Research Directors met to plan and discuss the program 



as well as the consultants to be summoned. The Steering Committee was 
composed of a member from each of six geographical regions of the nation 
Special care was given to promote the needs of each region as well as ea< 



city. 

As the Appendices to follow indicate, the concensus of participants’ 
opinions toward the effectiveness of the various programs was quite posi- 
tive in all cases. It is noteworthy that many workshop attendants wrote 
letters of both constructive criticism and praise. It is also important 



that, in many cases , participants later professionally contacted associates 
from other cities in efforts' to continue the exchange of information. Jn , 

Eact, most attendees cited that unstructured s time was a desirable facet : | 

if the seminars because of tlie oppprmjnity to freely explore the ideas and 
experiences of representatives from other cities ’ systems. In all seminary 

. -i 4 
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thcref ore , several hours were scheduled informally -- usually during the 
late afternoon or evening after attendants had worked together for a time. 
In the meetings of the Superintendents from the twenty- one cities, 
there was great support of the project. Clear indication was given that 
the training activities of the programs were not only beneficial to their 
staff but also to themselves. Reports of the staff seminars were over- 
whelmingly approved, and these reports helped to create the agenda for 
their own discussions. The project provided the groundwork, therefore, 
for improving the link between the research -evaluation departments and 
the decision-making administrators. 

This project obviously represented just a beginning in the effort 
to improve management practices in large school districts by bridging the 
gap between evaluative information and the decision process. A great 
deal remains to be done, and it must be done quickly because of the severe 
crisis which urban school systems face. The project was successful, how- 
ever, in that it did result in improved skills, and, more important per- 
haps, it focused the attention of major decision makers on the research 
and evaluation process and on improvements which are needed. It also set 
a pattern of cooperation among major school districts, the Office of Edu- 
cation, university staff , and other knowledgeable persons in the research 
and evaluation field. This pattern needs to be followed up quickly in 
more comprehensive and more intensive efforts. 
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Dr. Harvey B, Scribner 
Dr i Mark R. Shedd r 
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Dr, Clyde C. Miller 
Dr. Jack P. Hornback ; 

• m 7 

Dr . Thomas Shaheen 
Dr; Hugh C. Scott 



BOARD MEMBER 
Dr. Benjamin Mayes 
Mr. Robert Karwacki 
Mr. Paul R. Tierney 
Dr. George Blackman 
Mrs. Louis A. Mai is 
Mr. Daniel 0. Corrigan 
Dr. Marvin Berkeley 
Mr. James Perri.ll 
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Dr. Robert L. Doctor 
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Mr. George Ilutt 
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Mr * Frank Case^ • 

Mr. Malcolm W. Martin 
Dr. Gene French 
Mr. Alan H. Nichols 
Rev. James Coates 17 
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APPENDIX B 



Agenda o£ the Washington, B,G, 
Session for Research Directors 



Washington, D.C. 



RESEARCH DIRECTORS 



"Evaluation as an Information System 
for Project Development" 



"Evaluation of Multiple Project Programs" 
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evaluation as an information system 

FOR PROJECT DEVELOPMENT 



Wednesday, November 12, 1969 



9:00 - 9:30 A.M. Opening Remarks 

Introduction o£ the 
Executive Vice President 



FIRST SESSION: Dr. Robert Lahkton, presiding 

9:30 - 11:00 A.M. Status of the Discrepancy 

Evaluation Model 



11:00 - 12:30 P.M. Status o£ CIPP (Context Input- 

Process Product Model Evaluation 



12:30 - 2:00 P.M. Luncheon - Announcements 

SECOND SESSION: Dr. John L. Dayman , Jr., presiding 

2:00 - 3:30 P.M, Description/ Judgement Evaluation 

Model 

3:30 - 4:00 P.M, Break 

4:00 - 5:00 P.M. Panel Discussion of Strengths and 

Weaknesses Related to Three Concep- 
tualizations of Evaluation Models 

6:00P.M. Dinner 







Dr, Joseph L, Mazur 
Director, Division of 
Research and Evaluation 
Cleveland Public Schools 



Dr. Malcolm Provus 
Director of Research 
Pittsburgh Public Schools 

Dr, Howard Merriman . 
Director of Evaluation 
and Research 
Columbus Public Schools 



Dr. Robert Stake 
Professor of Education 
University of Illinois 



Drs . Provus , Stake , and 
Merriman 
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EVALUATION OF MULTIPLE PROJECT PROGRAMS 



Thursday, November 13, 1969 



THIRD SESSION: Dr. Samuel McClelland, presiding 

9:00 - 10:30 A.M. Issues Related to Evaluation of 

Multiple Programs 



10:30 - 12:00 P.M. Nature of Management Evaluation: 

Specific Problems Related to the 
Development of Multi-Project 
Evaluation Designs 



12:00 - 1:30 P.M. Luncheon 

1:30 - 2:45 P.M. Open time for attending the 

Research Council’s General Ses- 
sion on Research: "Research -- 

"What Should Be the Goals and 
Priorities of the Research 
Council?" 

FOURTH SESSION: Dr. Joseph Mazur, presiding 

3:30 - 5:00 P.M. Issues Related to Organizing an 

Evaluation System for Multi-Pro- 
ject Programs 



Friday, November 14th, 1969 

FIFTH SESSION: Dr. Malcolm Provus, presiding 

9:00 - 10:30 A.M. Practical Implications of Opera- 
tionalizing Selected Evaluation 
Models • ;• ' 

10:30 - 12:30 P.M. Planning the Training of Research 

Staff 7-Vl .1 l '.y " 



12:30 1:30 P.M. 
SIXTH SESSION: yl 
1:30 - 3:00 P.M. 



Lundieon 

Dr. Joseph Mazur , presiding 

Planning Training Sessions for 
Research Directors y : : . 




Dr, Karl Hereford 
Director , Program 
Planning, B1SE 

Dr. Philip Kearney 
Associate Superinten- 
dent, Michigan State 
Department of Education 
Bureau of Research 



Dr. Francis A, lanni 
Columbia University 

Dr. James Gallagher 
U.S. Office of Education , 



Dr. Richard Jaeger 
Former Chief of Evalua- 
tion for Compensatory 
Education Programs , BE3E 
Currently on leave from 
Stanford University 



Dr. James Jacobs 
Director of Res e arch § 
Evaluation, Cincinnati 
Public Schools • 
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Vail, Colorado RESEARCH DIRECTORS 



"Systems Viewpoint in Project Development" 
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The Research Council o£ the Great City' Schools 

Research Directors Seminar 
Vail Village Inn 
Vail, Colorado 
May 19-23, 1970 

SYSTEMS VIEWPOINT IN PROJECT DEVELQBffiNT 



Wednesday, May 20, 1970 

MORNING 
First Session 

Systems Approach in Dealing with R,D,D, and E Problems Dr. Desmond Cook 
Second Session 

Basic Systems Concepts. Dr. John Skalski 



AFTERNOON 
Third Sessi on 

Systems Tools and Techniques, I 
Fourth Session 

Systems Tools and Techniques, II 

Thursday, May 21, 1970 

MORNING 
Fifth Session 

Research Director - A Systems Manager 
Sixth Session 

Systems Thinking in Single Project Planning and 
Development 



9 




Dr. G. Trzebiatowski 



Dr. Cook and 
Dr. Trzebiatowski 



Dr , Cook 



Dr. Skalski 
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AFTERNOON 
Seventh Session. 

Systems Thinking in Single Project Evaluation Dr. Trzebiatowski 

Eighth Session 

“ Systems Thinking in Multi-Project Management 



Friday, May 22, 1970 



MORNING 
Ninth Session 

The Belmont Project From A Systems Viewpoint Dr. Cook 

Tenth Session 

Performance Contracting - A Project of 

Systems Thinking Dr. Rogers Barton 

AFTERNOON 
Eleventh Session 

Implications of Using The Systems Approach, For 

The Research Director Resource Panel 

Twclvth Session 

What Does It All Mean? , . 

Where Do We Go From Here? Directors Panel 
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-StfmiAIt PARTICIPANTS 
Vail , Colorado 



Dr. Larry Orcutt 
Fasearch & Development 
Atlanta City Schools 
224 Central Avenue 
Atlanta, Georgia 30393 
(404) 522-3381 

3ir» Orlando F. Fumo 
Baltimore Public Schools 
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Baltimore , Gary laud 21210 
(301) 467-4000 3463 

Margaret IT. Callahan 
Office of Program Development 
2893 TJashington Street 
Roxbury , Massachusetts 02119 
(617) 445-6912 

Mr , Claude Clapp 
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712 City Hall 
Buffalo, I Jot j York 14262 
(716) 342-4660 

Dr, Irving Brauar 
Operations Analysis 
Chicago Public Schools 
228 Worth LaSalle Street 
Chicago, Illinois 60G01 
(312) 641-3039 

Dr. Joseph L. Uazue 
Cleveland Public Schools 
1330 East Sixth Street 
Cleveland , Ohio 44114 
(216) 696-2929 

Dr. Arnold Ashburn 
Research & Evaluation 
Dallas Independent Schools 
3700 Ross Avenue 
Dallas, Texas 
(214) 824-1620 



Dr, Joseph Brsseinski , Director 

Research Services 

Denver Public Schools 

414 - 14th Street 

Denver, Colorado 

(303) 266-2255 3*97 

Dr, Robert S. Lankton 
Detroit Public Schools 
5957 Woodward Avenue 
Detroit, Michigan 40202 
(313) 833-7990 M2 301 

Dr . Howard Bowman 

Los Angeles City Schools 

450 North Brand /.venue 

Los Angeles, California 90012 

(213) 625-8921 

Miss Terry Bond 
Memphis City Schools 
2597 Avery Avenue 
Memphis , Tennessee 38112 
(901) 323-8311 

Dr. Robert Brownlee 
Milwaukee Public Schools 
5224 West Vllet Street 
Milwaukee, Wisconsin 53208 
(414) 476-3670 



Dr, R. W. Faunce 
Minneapolis Public Schools 
807 N, E. Broadway 
Minneapolis, Minnesota 55413 
(612) 332-4284 

Dr. Samuel D, McClelland (6 Mrs 
New York City Schools 
110 Livingston Street 
Brooklyn, New York 11201 
(212) 596-4045 
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Seminar Participants - Cent'd 



Dr. William C. Theimer, Jr. 

Philadelphia Public Schools 
Parkway at 21st Street 
Philadelphia, Pennsylvania 19103 
(215) 448-3573 

Dr. Malcolm S'rovua 
Pittsburgh Public Schools 
341 South Belief ield Avenue 
Pittsburgh , Pennsylvania 15213 
(412) 682-1700 

Dr. Gerald H, Iloeller, Director 

St. Louis City Schools 

Division of Evaluation and Research 

1517 S. Theresa Avenue 

St. Louis, Missouri 63104 

(314) 865-4550 

Dr, William H, Vogler 
San Diego City Schools 
4100 Normal Street 
San Diego, California 
(714) 293-4681 X406 

Dr. Harold L, Weeks 

San Francisco School District 

135 Ven Nose Avenue 

San Francisco, California 94102 

(415) 863-4680 

Dr. Mildred Cooper 

Office of Budget Research & Legislation 
Presidential Building 
Room 1013 

415 - 12th Street, N.W. , 

Washington, D. C. 20004 
(202) 347-6383 
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SEH1NAR CONSULTANTS 



Desmond L. Cook, Director 
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John Skalski 
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(614) 293-4934 
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Planning 4 Research Division 
Dallas Independent School District 
3700 Ross Avenue, Room 45 
Dallas , Texas 75204 
(214) 824-1620 



TITLE I CONSULTANTS 



Marvin Dawson 
Diane Schoenfelder 
Cornelius Butler 



HEA REPRESENTATIVE 



Glen Robinson 

NBA Research Division 



COUNCIL STAFF 

John L, Kayman, Jr. , Research Director 
Mary Lou Armiger, Research Associate 
Jerry Calendine , Research Intern 
Jack Stenner, Research Intern 
Edward Whitney, Research Intern 
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the research council of the great city schools 

Agenda for February Staff Training Session 
Rivermont Holiday Inn 
Memphis , Tennessee 
February 25, 26, 27 



Wednesday, February 25 : 

8:15 ' BREAKFAST -- Room to be Announced. 

Opening Remarks and Introduction of 
Resource Consultants 

9 * 15 ISSUES IN MEASURING AFFECT : 

General Introduction to the Session 
Topic 

10 : 45 Break 

11:00 SMALL GROUP SESSION: Begin Outlining 

Specific Problems Experienced in 
School Systems 

12:00 LUNCH 

1:00 GENERAL NATURE OF ATTITUDE FORMATION 
AND CHANGE: Emphasis on Problems 

. . ■ Identified by Participants and on_ 

Areas of Concern in Public Education 

I 3:00 Break 

3:15 SMALL GROUP SESSION: Continue Outlining 

Problems 

5:30 COCKTAIL PARTY 



Dr. John Hayman 



Dr. David Orr 



Participants 
and Staff 



Dr. Marvin Shaw 



Participants 
and Staff 

Participants 
and Staff 
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Thursday , February 26 : 

ff: 00 SPECIFIC ATTITUDE REASURING 

INSTRUMENTS : Focus on the 

Likert Technique 

10 : 30 Break 

10:45 GROUP SESSION: Practician on 

Developing a Likert Scale 

12:00 LUNCH 

1:30 GENERAL ATTITUDE MEASURING 

INSTRUMENTS : Focus on the 

Semantic Differential 

3 : 15 Break 

3:30 SMALL GROUP SESSION: Practician 

on Developing a Semantic Differ- 
ential Scale 

7:00 PRESENTATIONS BY RESEARCH STAFF: 

Experiences in Developing and 
Using Affective Instruments 



Friday, February 27 : 

9:00 MEASURES OF AFFINITY: Hie 

Sociometric Technique 

10:45 Break ■ 

11:00 SMALL GROUP SESSION: Determination 

of Ways the Sociometric Technique 
Might be Used in School Research 

12:00 Lunch 

1:30 REPORT BACK ? FROM SMALL GROUPS; 

GENERAL SESSION AND FINAL WRAPUP 

3:30 Session Ends 



Dr. Everett Rogers 



Dr. Marvin Shaw and 
Participants 



Dr. Bradley Greenberg 



Dr, Bradley Greenberg 
and Participants 

Miss Marion Kilbane 
Dr. John Temple 
Dr. Leo Weisbender 



Dr. Everett Rogers 



Participants 
and Staff 



Participants and 
Staff 
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RESEARCH TRAINING SEMINAR 
PARTICIPANTS ' 



Atlanta, Georgia 



Miss Gaya Barnard 
Atlanta” Public Schools 
Research a jig Development Division 
Atlanta, Georgia 



Baltimore, Maryland 



N. Craig Cutter 

Bureau of Instructional Res e arc:'. 
Baltimore City Public Schools 
2521 N. Charles St. 

Baltimore, Maryland 21218 



Boston, Massachusetts 



Marian -J . Ego 

Department of Educational Ir.no . a <_io 
and Measurement 
Boston Public Schools 
45 Myrtle Street 
Bos ton , Mass a thus etts 



John LcConte 

Dept, of Title I Programs 
2893 Washington Street 
Boston, Massachusetts 02115 



Buffalo, New York 



Dr. Douglas Houck 

Divi s i on o f Curri culuxn Devel opr ant 
and Evaluation 
Buffalo Public Schools 
City Kali 
Buffalo, i‘ev7 Yor.< 



Albert Thompson 

Division of Finance and Resear:." 
Buffalo Public Sciiocis 
City Hall 
Buffalo, Ivov: York 



Mrs . Hallie Fr ancies k •_ 
Cleveland Boarct of i - .cucstic. . 
1380 East 6th Street ; 

Cleveland, Ohio 44114 



3Q 



Cleveland, Ohio 
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Cleveland, Ohio 



Dallas , Texas 



Denver, Colorado- 



Detroit , 



Mi 



.qan 



Los Angeles, California 



Memphis, Tennessee 








Miss Marian Kilbana 
Division of Research 
Cleveland Public Schools 
1380 East 6th Street 
Cleveland, Ohio 44114 



Dr. Arnold Ashburn, Consultant 
Research & Evaluation 
Dallas Independent Schools 
3700 Ross Avenue 
Dallas, Texas 



John Temple 

Office of Planning, Research, & 
Budgeting 
414 - 14 til Street 
Denver, Colorado 80202 



George W. Jacobs 
Schools Center - Room 862 
50 S 7 Woodward Avenue 
Detroit, Michigan 48202 



Dr. Leo W ei sbende r 

Measurement & Evaluation 
. (Los Angeles City Schools ) 
£s810 Emerson Avenue, Room 7 
Los Angeles, California 90045 

Mrs. Virginia W. Blanton 
2597 Avery 
.Board of Education 
Memphis, Tennessee 38112 



Miss Terry Bond 
2597 Avery 
Doard of Education 
Memphis, Tennessee 38112 

Miss Kathy M. Eggers 

Memphis Community Learning Laboratory 
J/u s, Orleans 
Memphis , Tennessee 
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Milwaukee , Wisconsin 


Dr. Gary D. Peterson 
Milwaukee Public Schools 
5225 West Vliet Street 
Milwaukee, Wisconsin 53208 


Minneapolis, Minnesota 


R.W. Faunae 

Minneapolis Public Schools 
807 N.E. Broadway 
Minneapolis, Minnesota 55413 

Mrs. Rebecca Howard 
Minneapolis Public Schools 
807 N.E. Broadway 
Minneapolis, Minnesota 55413 


New York, New York 


Mr. Howard S, Tills 
Bureau of Educational Research 
110 Livingston Street 
Brooklyn, New York 


Philadelphia , Pennsylvania 


Dr. Robert G. Reiter 
Office of Research & Evaluation 
School District of Philadelphia 
21st Street at Parkway 
Philadelphia, Pennsylvania 19113 


Pittsburgh, Pennsylvania 

7 

% 


■ Glenn E . Qxteer 

Research Associate 

Pittsburgh Board of Public Educatio 
Office of Research 
249 North Craig Street 
Pittsburgh, Pennsylvania 


St. Louis, Missouri 

■ v ' 


. h. T upper Drans 

Division of Evaluation & Research- 
St. Louis. Public Schools 
1517 Theresa Street 
St . Louis ,. Missouri. • 


San Diego, California 


Mr. Stuart Macnofsky 
c f i n rr S erv l G ti s 


o '"t ; V" 

eric: 


San Diego City Schools 
■ 4100 Normal Street 

San Diego, California 



r 
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Ban Francisco, California 



Mrs, Mary Jane Fernandez 

ESEA Compensatory Resource Center 

844 Folsom Street 

San Francisco, California 



Washington, D.C. 



Mrs. Beulah G, Glenn 

Department of Research & Evaluation 
D.C. Public School System 
415 - 12th Street, N.W. 

Washington, D.C. 20 00 4 



Mrs, Josef in a M. Ordonez 
Department of Research & Evaluation 
(D.C. Public Schools 
415 - 12th Street, N.W. 

Washington, D.C. 20004 



Research Council Consultants: 



Dr. Bradley Greenberg 
511 S. Kedzio 
Michigan State University 
E. Lansing, Michigan 48023 



Dr.. David B. Orr 

Scientific Educational Systems, Inc. 
#607, 910 17th Street, N.W. 
Washington, D.C. 20006 



Dr. Everett Rogers 
Department of Communication 
Michigan State University 
East Lansing, Michigan 

Dr. Marvin Shaw 
Department of Psychology 
University of Florida 
Gainesville, Florida 32601 
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Research Council Staff: 



Dr. John Hay man , Jr. 

Research Director 
Research Council of the Great 
Schools 

1819 "H" Street, N.W. 

Suite #850 

Washington, D.C. 20006 



Miss Mary Lou Armigor 
Research Associate 
Research Council of the Great 
Schools 

1813. "II" Street, N.W, 

Suri-te ipBSu 

Washington, D.C. 20006 



Mr. Gerry Calendine 
Re sear ch I n t e r n 

Research Council of the Great 
Schools 

1819 "H" Street, N.W. 

Suite #850 

Washington, D.C. 20006 



Mr. Jack Stanner 
Research Intern 

Research Council of the Great 
Schools 

1819 " H " S tre e t , K . W . 

Suite #850 

Washington , D.C. 20006 



Mr. Edward Whitney 
Research Intern 

Research Council or the Grea.- 
Schoois 

1819 "K" Street, i\»W. 

Suite #850 

Washington, D.C. 20006 
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Memo of Suggestions 
Voiced at the Final Memphis Session 

1, Research Council might serve as a clearinghouse through which member 
school districts could share the following: 

a. Locally devised measurement instnnnents, with examples of 
their specific application; 

b. Computer programs, described via one -page abstracts; 

c. Optical Scanning ("Digitek") forms designed for specific 
uses in research and evaluation; 

d. Examples of good format for final reports to various 
audiences, such as a brief report plus separate techni- 
cal supplement, a concise and separately bound "execu- 
tive summary," and audio-visual aids for oral briefing 
of administrators. 

2. Research Council might seek to orient our superintendents and associate 
superintendents to the value, proper role, and appropriate use of research. 
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Training Institute for 
Research and Evaluation Personnel 
of the Great City Schools 

September 20-23, 1970 
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Report of Training Institute for 
Research and Evaluation Personnel 
of the Great City Schools 

Introduction and Background 

During the dates of September 21-23, 1970, Teaching Research 
conducted a training institute, in Monmouth, Oregon, for research 

i 

and evaluation personnel of the twenty-one cooperating School Districts 
of The Council of the Great City Schools. 

Preliminary planning and negotiations for the training institute 
had been negotiated with Dr. John L. Hayman, Jr., Director of Research 
of The Council of the Great City Schools during tfje spring of 1970. 

(See Attachment A) It was agreed that a three day institute would be 
held during late September, 1970, in Oregon for research and evaluation 
personnel of the Great City Schools Research Departments to be organized 
around the four areas of proposal writing, instructional systems, measure, 
ment and evaluation# 

It was further agreed that the following details would be Included 
in the institute i 

X, A pre-test to assess the competence level of each participant 
before instruction, 

2. Independent learning activities, adjusted to each individual's 
needs, to the extent possible. 

3. A post-test to measure growth. 

4. Certification as to what each participant was able to do 
in relation to his selected area of study. 

5. An evaluation of the effectiveness of the institute, copies of 
tjhich would be sent to each Research Director. 



<3 
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Psrticipants In the institute numbered thirty-two. The Council 



of the Great City Schools assimed the responsibility for their selec- 



tion and notification. Arrangements for transportation and housing 
were handled by Teaching Research, All participants were met at the 
Portland .International Airport and motored 50 miles south to Salem 
where they were lodged at the Marion Motor Hotel, Participants were 
assigned to cars by groups and driven 15 miles to the Oregon College 
of Education campus in Monmouth each day. The list of participants 
is included as Attachment B. 
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Obleetives 

The purpose of the Institute was to bring research and evaluation 
personnel of the Great Cities research departments together to; 

1. Provide instruction in the areas of measurements, evaluations 
.instructional systems* and proposal writing. 

2. Provide an environmental conducive to independent, learning 

. activities and adjusted to each individual’s needs; and 

3. Provide and assessment of the competence level of each 
participant in relation to his area of study. 

In addition to the above* terminal behaviors expected from each 
of the four areas of instruction were defined and negotiated with 
each participant and his instructor (See Attachment C for detailed 
objectives in each area). 

Program 

An outlined summary of the institute schedule has been Included 
as Attachment D. In general, the instructional sequence minimized 
passive mass-reception and maximized active individual and small group 
involvement. The institute was initiated Monday, September 21 * 1970 
at 8*30 p.nu, with an orientation during which participants and staff 
were introduced and specific goals and procedure of the institute were 
clarified. 

The total group was then divided into four subject groups* 
Participants were given the choice of joining any one of the following* 
evaluation, instructional systems, measurement or proposal writing. 

The group assembled in separate areas mid spent the remainder of the 
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morning being pre-tested and negotiating individualized performance 
goals for each member. The latter was accomplished by the instructor 
of each group sitting with each member of his group to (1) review the 
results of that persons post-test. (2) clarify for the member, any 
questions, he might have concerning the behavioral objectives established 
for that subject group, (3) identify specific interests of .that member 
in the area, and (4) negotiate a set of behavioral objectives with 
that member which took into consideration the members pre-knowledge 
of the area, his expressed interests in the area and reasonable limits 
of departure from the goals established for the institute. 

From these negotiations, individualized learning experiences 
were then planned for the remainder of the institute time period. With 
the exception of the final total group meeting during the late after- 
noon of the last day of the institute, the institute was devoted to 
independent learning activities* 

Two "extra" learning activities were also planned and carried out 
during the institute. Noon luncheons were scheduled in a convenient 
local restaurant with provisions for all other staff members of Teaching 
Research not involved in the institute (approximately thirty) being 
present , 

Several lasting acquaintances between participants and Teaching 
Research personnel have emerged from these informal interactions with 
resulting benefits to both The Council of the Great City Schools and 
Teaching Research. 

The second extra activity involved the transportation of institute 
participants and staff to the Oregon Coast on the afternoon of the 
second day to relax and enjoy the fall beauty of the rugged coast and 
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t© partakes of a variety of excellent seafood. Although no learning 
experiences directly related to the goals of the institute were 
planned as a part of the coast visit, obvious indirect benefits 
did result as noted in subsequent expressions of pleasure and satis- 
faction from participants and a generally more relaxed altitude on 
their part during the remainder of the institute. # 

Evaluation 

The principal evalaution of this institute was in the form of 
post-tests within each subject group. Since each individual 
participant negotiated his own specific set of behavior objectives, 
a formalized Institute post-test was not appropriate. Instead, each 
subject area instructor carried out post-testing to fit the particu- 
lar needs of his participants. With the exception of one member who 
had to leave on the second day of the Institute and two other members 
who were from the Council's Central Office who had continuous inter- 
ruptions which prevented their maintaining instructional continuity , 
each participant was judged as having achieved at least the minimal 
level of performance he had established for himself. 

In addition to the above, the final debriefing of all participants 
was conducted during the final hour of the institute. The group was 
asked to respond to the following three questions: 

1. What, would have improved the institute in helping you to 
accomplish your learning goals? 

2. What "warm blanket" treatment was not gcod? 

3. Did you accomplish what you expected from the institute? 

Responses to these questions are summarized as follows: 
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Item I: 

—It would have been better If institute instructional materials 
could have been distributed to participants before coming. 

—More instructured interaction time among participants would 

have, been beneficial. 

Pre-testing of participants via mail prior to the institute 

would have given the institute staff a better opportunity of 
preparing for individualizing instruction. 

—More time to sit down individually with institute staff would 
have helped. 

Item 2: 

—"My kidneys are killing me from riding in that bus to the coaat. 

"It would have been better to stay In Monmouth rather than 

Salem and save the drive." 

— "We should have gone to the coast sooner." 

Item 3: 

— "My expectations were met," 

—"The time was too limited, it should have been a week." 

—Small group discussion of common problems was excellent but should 

have been increased. 

— Many expressed strong interest in having a second institute 
scheduled to follow-on as soon as possible. 

Recommendations 

The following recommendations were compiled from the debriefing 
remarks, general conversations with participants and reactions from 

the staffs ■ 

42 



O 

ERIC 



-40- 



1 , Pre-institute assessment of participants relative to the 
objectives of the institute should be accomplished. Not only would 
this afford institute planners a clearer understanding of the partici- 
pants and how they differed, it would allow more realistic planning for 
individualization as well as the gathering of special learning resources 
which might be useful for participants. ' 

. 2. Instructional experiences should be prepared such that they 

are approached from the point of view of their relevance to large city 
school districts. Rather than utilize abstract examples or situations 
which are very distant from large city school problems in the instructional 
materials, direct translation to current problems being encountered in 
the urban school districts should be prepared, (Of course, the latter 
cannot be accomplished without considerable lead-time for preparation.) 

3. The listing of institute goal statements and objectives along 
.with the basic reference materials to be used, should be mailed to each 

participant at least two weeks before the start of the institute. By 
Introducing participants to the general concerns of the Institute and 
informing them about available resources, participants could arrive with 
more realistic goal expectations and a more uniform entry information level 

4. All individualized learning materials, e.g., slide-tapes, films, 
etc., should be made available early in the institute so that participants 
would have the advantage of being introduced to the institute's philosophy 

and orientation to specific issues, 

5. A cluster of rooms in the hotel in which participants are lodged 

should be obtained to provide easy group gathering during evening hours to 
permit discussion of pertinent topics as to facilitate various social 
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activitles which allow discussion of common problems. 

6, Small group discussions of relevant topics, using partici- 
pants as presenters, should be utilized and scheduled very early In the 
institute. The participant group is usually a .powerful one in that they 
have first hand knowledge of a variety of problems in their jobs. Many 

i 

of the solutions to problems devised by these people are often excellent 
and can prove to be a useful mechanism to involve the group and extend 
into the Institute subject area. 
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Attachment A 

s 



Hay 29, 1970 



Dr. John Hayman 
Director of Research 
Research Council of the 
Great Cities Schools 
1818 H, Street N.W. 
Washington, D.C. 20006 



Dear Dr. Hayman: 

It is my understanding, as reported to me by Jack 

Great Citie7 Research Council wishes Teaching Re ® e ^ C ^ h e° G °e!t Citie8 
conduct a training session for research personnel of the Great Cities 

research departments. The institute will be held in Oregon in lat 

September over a three day time period. 

We propose that the institute be held during the dates September 
‘ 21 - 23 %ith participants arriving on Sunday, September 20 and d «P«ting 
early Thursday, September 24. The site would be on the Oregon coast, 
either at Salishan Lodge or The Inn at Spanish Head. 

The institute will be organized around the four areas of 

writine instructional systems, measurement, and evaluation. Participants 

Research Directors . 

The following details will be included in the Institute: 

1. A pre-test to assess the competence level of each participant 
before instruction. 

2. Independent learning activities, adjusted to each individual's 
needs, to the extent possible. 

3. A post-test to measure 
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4 . Certification as to what each participant is able to do in 
relation to his selection area of study, (X,f a participant 
completes his first area of study before the institute con- 
cludes , he may begin a second topic of study.) 

1 9 

5. An evaluation of the effectiveness of th institute, copies of 
which will be sent to each Research Director, 

• 

The costs Involved in organizing and conducting the Institute are 
as follows: (these are based on 30 participants and four staff members 

for a three day institute) 



Instructional staff, $450/staff member x 4 $ 1,800* 

Support services and planning time, $375/staff member x 4 1,500 

Travel, SORT - Portland to Salishan * 180 

Student drivers, 150 hrs, @ $2 *00 /hr, 300 

Participant instructional materials $15 x 30 A 50 

Subtotal $ 4,230 

Indirect costs - for contract processing, 
other administrative overhead, equipment 

and facilities @ 8* x $4,230 

*Employee benefits are included in this figure TOTAL $ 4,568 



Each additional participant beyond 30 should be budgeted for $100 which 
includes in-state travel to institute site, instruction, and materials. 

In addition to the above costs, you of course will have to budget for 
individual travel to Portland and per diem. As soon as we nave confirmation 
of the number attending we will make lodging reservations anu inform you of 
details such as room and food costs. 

I hope you will find it convenient to come to Teaching Research in the 
near future to work with us in preparing a proposal aimed at continuing the 
training of research personnel of the Great Cities Schools* We are con— 
siderably enthused about such a possibility and look forward to working with 
you, 

*■. 

Jack Edling asked me to express his appreciation for the opportunity of 
meeting in Denver with the Research Council of the Great Cities Schools, to 
clarify the details of this agreement. I am writing this letter to indicate 
that this is a firm contractual arrangement between the Great Cities Research 
Council and Teaching Research. 



Sincerely yours , 



Dale G, Hsnreus 
Associate Director 



DGH:ss 

cct Dr. Jack V. Edling 
Enclosures 
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Attached are four areas related to research in which training will e 
offered in the fall. After reviewing each behavioral description, complete 
the form below and return to : s 

Dr. John Hayman 
Director of Research 
Research Council of the 
Great Cities Schools 
1819, H. Street N.W. 

Washington, D.C. 20006 



Name 

Pos i Lion 

Institutio n 

Highest degree attained 

Major area of preparation 



city 



State 



Indicate your preference of areas 
training by placing 1 in front of your 
second choice. 



in which you desire to receive 
first choice and 2 in front of your 



Proposal Writing 
Instructional Systems 
Measurement 
/Evaluation 
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Proposal Writing - After completing the package on proposal writing, the 
laartier will: 



1. identify major componenets of a proposal, their function and 
criteria for determining their adequacy by demonstrating from 
recall, an ordered set of proposal components. 

2. detect and prescribe a remedy for certain typical J* 

proposals by identifying weaknesses in examples and indicating 
changes required to reach acceptable standards, 

3 construct a sound proposal outline in a problem area of personal 
interest and write a proposal which meets funding s tan ’.rds a* 
determined by experienced proposal viewers from Teaching Research 
who are currently reviewing proposals for the U,S. Office of 
Education . 



% 
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Instructional Systems - After completing the package on instructional 
systems, the learner will : 



1, identify what is meant by the systems approach to instruction 
by defining systems and management elements, detailing design 
and analysis strategies and specifying development and assess 

*taant procedures - 

2, specify instructional sequences by demonstrating a method of 
objective analysis that guides the designer In determining 
what enabling objectives are prerequisite to terminal objectives 
and the order in which each objective should be taught, 

3, specify Instructional conditions by identifying a series of guide 
lines that detail learner characteristics, the instructional 
context. Instructional stimuli, learner responses, and feedback 
routes . 

4 define the relationship between research and the instructional 
development by identifying distinguishing characteristics. 
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Measurement — After cumpleting the package on measurement , the learner will* 

1, derive, in logical fashion, measurable characteristics from 
educational objectives having a range from concreteness to ab- 
stractness. 

2. identify the scale (i . e . , nominal, ordinal, interval, or ratio) 
represented in any measuring instrument considered for use and 
express in writing the limitations each scale imposes on the 
meaning of the obtained data. 

3* define, on a matching basis, key terms involved with reliability 
and validity; express examples of each; and specify ways of 
reducing error in measurement. 

4 define key concepts which tend to reduce issues of reliability 

and validity in measurement and cite independent examples of each. 

5. reflect familiarity with various classes 6f measures by discussing 
issues involved in the use of specific ins t rumenls . 

6. construct measuring instruments that are valid for the purpose cf 
measuring stated instructional objectives. 
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Evaluation - After completing the package on evaluation, tie learner will 



1 . establish for himself the purpose for evaluation by answering 
questions such as '’who needs what information? When is the 
information needed? What classes of decisions are to be made? 

2 . define the context within which evaluation is to be conducted 
which includes ider ti f ication of variables to be evaluated, clari- 
fication of the role and use of objectives and objective speci- 
fications in identifying value parameters, and procedures for 

the identification of criteria in setting standards for evaluation. 

3 . cope with evaluation decisions from the point of view of a program 
director which include? an understanding of several leading eval- 
uation models , strategies for evaluation design and need assess- 
ment strategies . 

4 . identify the principle features of the tools of evaluation which 

include: ( 1 ) information collection procedures (sources of needed 

information, form in which information will be needed, decision 
criteria for determining collection procedures, sampling): ( 2 ) 

instrumentation (standardized tests, judgmental responses, question- 
naires , interviews , observational methods, unobtrusive measures , 
instrument credibility); (3) information processing (data t_ pes , 
purpose of analysis, levels of measurement, data organization, 
statistical treatments, displaying results); and (4) Information 
distribution (audience characteristics, single or multiple distri- 
bution channels, purpose of distribution). 
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Attachment B 

Great Cities 1 Institute 
Teaching Research 





.September 20—23, 1970 


■NAhffi 


TOWN & STATE 


Mary Lou Armiger 


Washington, D. C . 


Phillip Bolger 


New York, New York 


Terry Bond 


Memphis, Tennessee 



Kenneth Bourguignon Atlanta, Georgia 



Jerry Calendine 


Washington, D-C. 


James Carpenter 


Chicago, Illinois 


John LoConte 


Boston, Massachusetts 


H* Graig Cutter 


Baltimore Maryland 


William Denton 


Washington, D.G. 


Arthur Draper 


St. Louis, Missouri 


Roger Fish 


Washington, D.C. 


Arthur Flater 


Milwaukee , Wisconsin 


Orlandon F- Furno 


Baltimore , Maryland 


John Hayraaa , Jr. 


Washington, D.C, 


Wilford Howard 


Denver, Colorado 


Larry Johnson 


Minneapolis , Minnesota 


Marcella Kirk 


Chicago, Illinois 


Michael La Bay 


San Diego, California 


John Lindsey 


Detroit, Michigan 


Richard McMenemy 


Portland, Oregon 


Joan O' Halley 


Chicago, Illinois 


Sam Mason 


Philadelphia., Pennsylvania 
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NAtffi 


TOWN & STATE 




Vincent Piraino 


Milwaukee s Wisconsin 




John L . Posa 


Los Angeles , California 




Glenn Queer 


Pittsburgh s Pennsylvania 




Jack S tenner 


Washington, D.C. 




Derek B. Taylor 


Cleveland, Ohio 


- 


Albert Thompson 


Buffalo, New York 




Melvin Tldyman 


San Francisco, California 




Lavolia Vails 


Washington, D.C. 




James H s Van Orden 


Washington, D.C. 




Bill Webster 


Dallas, Texas 
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Attach^ent C 

Great Cities Institute 
Teaching Research 
Monmouth, Oregon 
September 21-23, 1970 

Introduction ; The Great Cities Institute has brought research 
personnel of the Great Cities research departments together to: 

1, Provide instruction in the areas of measurement, evaluation, 
instructional systems and proposal writing; 

2, Provide an environment conducive to independent learning 
aetivlttel and adjusted to each individual needs ; and 

3, Provide an assessment of the competence level of each 
participant In relation to his area of study* 

Each participant, at his own choice, will receive instruction in 
one of the following four areas. Expected outcome behaviors are 
defined but will he subject to modification for any individual as a 
result of negotiated changes with his subject area instructor. 

1, Measurement (Dr. James Beaird). After completing the package 
on measurement , the learner will: 

Bm derive, in logical fashion , measurable characteristics from 
educational objectives having a range from concreteness to 
abstractness, 

b. identify the scale (i.e., nominal, ordinal, interval, or 
ratio) represented in any measuring instrument considered 
for use and express In writing the limitations each scale 
imposes on the meaning of the obtained data. 

c. define, on a matching basis, key terms involved with 
reliability and validity ; express examples of each; and 
specify ways of reducing error in measurement . 

d . define key concepts which tend to reduce issues of reliabi- 
lity and validity in measurement and cite independent examples 
of each * 

e. reflect familiarity with various classes of measures by 
discussing; issues involved in the use of specific Instruments* 

f* construct measuring Instruments that are valid for the purpose 
of measuring stated instructional objectives* 



O 
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2. Evaluate on (Dv. Frank Nelson). Upon completion of three days 
-study in ovaludtion, a participant in the Great Cities Institute 

„ will r,e able to create an evaluation design for a specified 
project which will provide both adaptive and descriptive infor- 
mation congruent with established ( or given) value standards. 

3. Instructional Systems (Dr, Floyd Urbach) . After completing the 
package on instructional systems, the learner villi 

I 

a# what in raean t by the systems approach po instruction 

by defining systems and management elements, detailing design 
and analysis strategies and specifying development and assess 
nent procedures . 

b. specify i ns truct ional sequences by demonstrating a method of 
objective analysis that guides the designer in determining 
what enabling objectives are prerequisite to terminal objec- 
tives and the order in which each objective should be taught. 

c. specify instructional conditions by identifying a series of 

guidelines that detail learner characteristics, the instruc- 
tional context, instructional stimuli, learner responses, 
and feedback routes * * 



Proposal Writing (Mrs. Lee Green). 

Specific Knowledges to be acquired: 

a. To be able to recall and demonstrate understanding of 
the function of major components of a research proposal. 

b. To demonstrate ability to locate information on fundi na 

' sources , . 6 

Analysis skills to be developed: 

c. To be able to identliy weaknesses and strengths in 
component pares of a research proposal. 

Synthesis skills to be developed; 

d. To Improve and correct component parts of a research 
proposal by rewriting identified areas of weakness. 

Synthesis and Evaluative skills to be developed: 

e. To create an outline for an adequate proposal in the 
student’s area of particular interest. 

f • To evaluate a research proposal. 



Daily tralnsportation between the Marion Motor Hotel and Teaching 
Research will be provided participants. Pick-up will be at 8 a a at 
the covered guest entrance of the Hotel. Departure from Teaching 
Research will be 5:00 p.m. in8 
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Attachment D 
INSTITUTE SOiEDULL 

Monday • • September 21 

8:00 a.nw Depart for Teaching Reaearch 



8:30 a.m, 
9:00* a .m • 



Introductions and Orientation 



Dale Hanreus 



Qr . Jim Beaird 
Dr* Frank Nelson 
Dr. Floyd Urbach 
Mrs • Lee Green 



(Coffee 
12 noon 

1 : 30 p* m* 
5:00 p*m. 



Break into subject groups: 

Measurement 
Evaluation 

Instructional Systems 
Proposal Writing 
rentes ting 

Each participant negotiates a specific set of outcome 
goals to be achieved during institute* 

in TRAC each morning and afternoon as desired,) 

Lunch: Blue Garden (walk) 

Interaction with other available members of Teaching Research 



Independent learning activities 
Depart for Marion Motor Hotel 



Tuesday - September 22 

8:00 a.T.. Depart for Teaching Research 

3:30 a .m. Independent Learning Activity 

12 noon Limch: Blue Garden 

1:30 p.m. Independent Learning Activity 

3:30 p.m. Depart for Oregon Coast and Salishan Lodge for dinner 
8:00 p. m, (approximate) Depart for Marion Motor Hotel 



8: 00 p.m # 

Wednesday - 


.8:00 


a . m , 


8:30 


a * m . 


12 noon 


1:30 


p.m. 


3:00 


p.m. 


o 

o 


p .B. 


O :00 


Pi3* 



Depart for Teaching Research 
Independent Learning Activity 
Lunch; Blue Garden 
Indep* learning Activity 
Pos t- -v .; % v;.;;\d Debrief 

Group Meeting: 5 minute report from each group 

summarizing accomplishments 

Departure for Marion Motor Hotel 
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APPENDIX F 



Sample of Reactions 
to Programs 
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DIVISION OF 

research and development 



ATLANTA PUBLIC SCHOOLS 



administration building 



324 CENTRAL AVI,, S.W. 



ATLANTA, GEORGIA 30303 



October 19, 1970 



Dr, John Hayman 
The Research Council of the 
Great Cities Schools 
1819 H Street* N. W. 

Washington, D. C. 20006 

Dear Dr. 1 layman: 

As one of the participants in the Great City Schools Teaching 
Research Workshop held at Monmouth, Oregon, I believe this letter of 
appreciation is in order. 

It has now been three weeks since I have returned to Atlanta and it 
seems that every day I have been able to make use of something I brought 
back from the workshop. I feel this indicates a high degree of relevancy 
and utility in the materials presented. Although the workshop at 
Monmouth was only the first one I have attended, if it was typical 
of the quality of work being done by the Research Council of the 
Great City Schools, I heartily endorse anything you may plan for the 
future and look forward to being in attendance if at all possible. 





Kenneth Bourguinon 
Research Assistant 




cc: Dr. Alvin G, S kelly 
KB/gl 





q / " 
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t>!VlfiirN Q~ EVALUATION 

and Hess arch 



Dr* Robert G* Reiter 
Division of ? ■ search and Evaluation 
Phil '.dolphin public Schools 
Philadelphia, Dennsyl vania 

Dear Dr. Reiter: 



i'rtrch A, 3 57 0 



iiv 












co^ 



I ci very happy to hava had the opportunity to meet you at the recent 
se’tair.ir in lAr-his sponsored by the Research Council of the Croat 
Cities Schools. I an equally glad that I vns in th • small fvoup d-.r-- 
cucsicn with you in which you explained the forced < ;;oice toeio-mcuvic 
instrument tl-.at you have developed for use in Philadelphia* Although 
I do not have a copy of the in trument , 1 was able *:o take adequate 
note to explain it: to other members of cur division., 



In the lest session of the 'seminar, on Frio. ;/ af Lr.-.n.wm, yo pay 
recall that ;V, Ksyu'-;. described a system of key in.. ’.cat or r; that is 
bein': developed for u: e in Philadelphia, 312 msntic : . :t trial thirty* eight 
vcriabl ee wer -. considered originally but to -it the }. \rt ban been reduced 
to treaty 65 If event variables , ec:..G of which would to gathered monthly 
and some anr.ually. Vie have br.cn studying y-.vopor&tc, y to dr-.volopmont a 
pirn 1.1 nr system to be used with the St, Louie Public: hcnooj.r * We £’re 
currently at the stage of trying to identify varieties flu..: will be 
adequately informative to justify the cost of inclu.'iovi in the infor- 
mation system. Pcrhe* 3 you could share seres of your rindmos in tuis 
respect with us. Could you send us a list and description of the 
variables that you pirn to include in your system. A:; well ■■« those 
that have been set aside. If you have any statist in.’-.l iir.nlngs, 
technical memoranda, or published materials on this project, we would 
be ve*>-y interested in studying them. /. • 

It vas wonderful to have met you at the sdm.Iviai* in V vmphifi. X w?.ll • 
be looking £ orward to .meeting' you again at some fut- re data , ;• . 



Sincerely yours , 




H, SUITER DRAKE 
Adminis tr& t ive.. Ass ist - ;«t 



IlTD/lb 
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OCT 2 6 REC'D 



DENVER PUBLIC SCHOOLS 

414 FOURTEENTH STREET / DENVER, CO K« 
HOWARD L, JOHNSOV, Sut^rtnirndi’nt 




OFFICE OF PLANNING. RESEARCH. AND BUDGETING 

JOSF.H! F„ BHZFlN'SRf. Far, uhv* Dirrtiur 

October 22* 1970 



Dr, Alvin G. Skelly, Executive 
Vice President 
The Research Council of the 
Great Cities Schools 
1819 H. Street M.W, 
Washington* B.C, 20006 



Dear Dr. Skelly: 



The Great Cities Research Council is to be commended for 
their efforts in promoting sound educational research 
practices. 



The personnel of the. Denver Public Schools Research 
staff who have been fortunate enough to attend the 
Great Cities Research Conferences have returned inspired 
and singing the praises of Dr. John I layman and his staff* 
Rarely does one find the opportunity for intellectual 
and social interchange* between school representatives 
on a national level. The activities of the Council nave 
made this possible* 

The workshop at Monmouth*' Oregon demonstrated the 
organizational expertise of the Council staff * Arrangements 
were exacting f activities timely* and relevant. The 
enthusiastic leadership of the conference was ^contagiou^ 
affecting all participants. 

Today we are witnessing a greater select ivity v 

part of funding agencies. An under takipg^must be - ^ 
educationally sound* have practical value* and be capable 
of ef f ecting desirable change. The activities of theory 

Great : ! -'Cities' Council certainly qual^i^ all these;; areas v 
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Dr, Alvin G, Skelly 
•October 22, 1970 
Page #2 



If educational practices and procedures are to change 
it will be through projects such as the Great Cities 
Research Council. School district representatives from 
across the nation must be brought together to share 
knowledge, to exchange ideas, and to develop some common 
dialogue relating to educational needs in large 
urban areas. 



The Planning, Research, and Budgeting staff of the \ 

Denver Public Schools supports the objectives and activities 
of the Great Cities Council and extends to Dr. John Hayman 
a sincere vote of confidence for a job well done. 



Sincerely, 





t 



Will Howard, Supervisor 
Office of Planning, Research, 
and Budgeting 



WHi cd 



ccs Dr, John Hayman 
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BOARD OF EDUCATION of THE CITY OF NEW YORK 
bureau of educational research 

1 10 LIVINGSTON STREET 
BROOKLYN p N. Y. 11201 



10/9/1970 



SAMUEL D. MCCLELLAND 
ACTlNO DIRECTOR 

GEORGE FORLANO 
Assistant administrative: director 



Dr, John Hayraan 
The Research Council of the 
Great Cities Schools 
1819 H, Street N.W, 

i, D.C. 20006 



Dear Dr. Hayman: 

Congratulations upon the excellent Job you 
Four years ago I attended 

but notice the vast iraproveme * al efforts made for Northwestern but 

years. This is not to demean the Gne ouraging to those of us who 

merely to note an obvious trend. It 9 ro c B ai , hh t© see the Council 

"labor in the vineyards" of urban euca_io^a participating cities. Few out- 

So/SS^^r^he of motivation and idea., not to mention the 

catharsis provided by the experience itself. 

X was oost favorably tap*"*- by' 01 ” 
of our hosts in Oregon, ^heir xelatx^ P . the usual abrasive social 

material* 

It is the general opinion of the three h£° Tili‘ and* "ysiif ) 

the conferences representing New Yorw Ci*y (Dr. - ’ , s evidenoe lest anyone 

constuuf^ thlfleitlf asTbiased ^plefSe all look forward to another meeting 

next year. 

It is my sincere wish f great^dca SplementefanS 

for this national achievement. I think we na y educational research. 

xo brin9 this 

to fnuiiion and^, once again, thanks. 



Copy: 



Dr. A.G. Skelly 
Dr. S.D, McClelland 




0 

M c 
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PORTLAND PUBLIC SCHOOLS 

631 Northeast Clackamas Street / Portland, Oregon 97208 
Phone (503) 234-3392 

EVALUATION DEPARTMENT 



Victor W. Doherty 
Assistant Superintendent 

Clifford W. Williams 
Director 



October Hu 1970 



Dr. John Hayrnsu 
The Research Council of the 
Great Cities Schools 
1S19 II. Street M,V. 
Washington, 0,0, 20006 

Dear John; 



I«d like to belatedly express my appreciation and support for the 
recent meeting we had at Monmouth, Oregon. 

Professional growth is a difficult thing to quantify, but I believe 
al l the participants would agree that a great deal of growth did 
occuri partly through the highly individualized program provided 
by Oregon College of Education and partly through the chance to 
exchange ideas and strategies for the solution of problems with the 
other confreres ■ 



Needless to say, X hope the possibilities for future meetings will 
receive serious consideration* 

1 ours truly. 




■'V RAI: ids 

cc ; Dr. Alvin G. Skelly 

" executive Vice Pr e sident 
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SAN DIEGO CITY SCHOOLS 

education center 

PARK ANP EL C WON BOULEVARDS 

SAN DIEGO 3. CALIFORNIA 



STUDENT SERVICES division 



October 14, 1970 



Dr, John Hayman 
The Research Council of the 
Great Cities Schools 
1819 H. Street N.W. 
Washington, D,G, 20006 



Dear Dr, Hayman: 

Will Howard of the Denver Public Schools has asked me to relate 
to you my impressions of the September workshop in Monmouth, 
Oregon. Since we have discussed the evaluation section during . 
your visit to San Diego this month I can only reiterate my feeling 
that the benefits derived from the three days at Monmouth were 
well-worth the time spent. The sharing of ideas among evaluators 
and the expertise provided by the Teaching Research staff combined 
to make a pleasant and productive experience. Speaking for my 
school district, it is our opinion that such a project is worthy 
of continued support. 
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THE SCHOOL DISTRICT OF PHILADELPHIA 
BOARD Or EDUCATION 
215t STREET SOUTH OF THE PARKWAY 
19103 



^EARK R. SHHDD 

SUPEXX.>nTN»5NT OF SCHOOLS 

WILLIAM C. THELMER, JR. 

OT.CVTrvH D' =-ECiOft 

AND L VALUATION’ 



JOHN B s PSPER 
director . 

443=3755 

March 17 s 1970 Edward k. brown 

assistant oipectoa 
448 -3* -li 



Mr. H. Tupper Drane • 

Division of Evaluation and Research . 

Saint Louis Public Schools 

•1317 U. Theresa Avenue ‘ . 

Saint Louis, Missouri 63104 • 

Dear Mr. Drane i 

• am enclosing throe items? v?hich I hope will be of’ some help as, you consider 
the expansion of your information system: 

(1) A list of the 20 key indicators; 

Definitions of soma toms used in that list; 

(3) A list of 13 optional indicators. 



Because of financial restrictions, the key indicator program, has been only 
partly .implemented to this date. Although we arc* hoping for gradually inursnsing 
XL-picmentation area year to year, our experience in using the indicators has been 
so • limited that ?>a casino t announce any conclusions at this time. However, I can 
say that we have not found any reason yet to doubt- the desirability of full 
implementation. . , , 



T shaj.e your enthusiasm about the Memphis seihinar and have told Dr. Kayroan 

so. . Among the factors I found especially exciting are (1) the personal associa- 
tions, (2) seeing that most of us are in the midst of a ' developmental, process' 
rather than loaded with conclusions, and (3) the subsequent opportunities : such as 
hhiu to share ideas- - .... 



, . X h ?”'® ^ .'enclosures' will; ; bs .useful, to you, and that wo may .mb.e-t again. 
I ' ll 3:a : interested to hear about: your progress.; with key indicators.. . • : . 



Sincerely yours r 







Robert: G. Reiter : > '.p\ 

•f-P ^ ... -rV-.T-' ‘ 

.• •Tes.t^vSpecialis-.t-.; -Sn -•?*: 

Office of h&oqarch . and V'Evaluatiion': 



RGRidmw 

Enclosure 







eg : v qDi v . a Joh n 

Dr. Hillium 



■ m* 






>hn L. Hay man, -Jr..'' • . - • --Dr. John • B . Pepc r ' 65 

.Ilium C. ihe.iinor, 2r. Mr.. ..Daniel R. Feibciono 

~ — . . _ : I .t 1 * 






An Eguaf Opportunity Employer 

Minneapolis Public Schools 

SPECIAL SCHOOL DISTRICT NO. I 

SCHOOL ADMINISTRATION BUILDING 
SOT Northeast Broadway Minneapolis, Minnesota 55413 



RESEARCH, DEVELOPMENT, AND FEDERAL PROGRAMS 
(012) 332-5371 

Don am D, Bsvw, Assistant Superintendent 



research division (612) 336-178S 

Richard W. Faunce, Consultant 

PiANNDTG AND 

DEVELOPMENT DIVISION (612) 332-537 1 
Lawrence P. Moon, Director 
Mary C. Kasbohm, Assistant Director 



October 12 , 1970 



Dr, John Hayman 
The Research Council of the 
Great Cities Schools 
l8l9 H Street N» W. 
Washington , D, C. 20006 



Dear Dr, Hayman: 



This brief letter is being sent to give you some feedback 
regarding the recent workshop at Monmouth, Oregon, sponsored 
by the Council of the Great Cities Schools. The materrals 
available through the Teaching Research Division oi the Oregon 
State System of Higher Education was very relevant to my needs. 
One of the most beneficial aspects was an increased knowledge 

* tt .... tinis 



of various educational evaluation systems. Hopefully. 



OI various - . . ' 

knowledge will result In the output of more productive research 

from this writer. 



Another benefit of the workshop was the contacts made with 
participants from other cities. We all agreed the sharing 
information and ideas was stimulating and should be continued. 

It appeared that all participants thought much could be gamed 
by additional workshops in this area, possmly concentrating _ 
more on specific problem solving by groups and increased sharing 
of information between cities. 



X believe use of additional funds for this purpose would 
be a good investment. 



Sincerely, 




Larjt/Johnson 
Research Associate 



A 



LJ:dm 



cc: Dr. Alvin G. Skelly 
Dr. R. W. Faunce 
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The Board of Public education 

Pittsburgh. Pa. ISSI3 

ApHlNlSTKiTlOH BUlLDIMQ 

BiLueni't and reReea avenues 



Dr. John L. Hayman 
Research Director 
Re search Council of the 
G.re at City Schools 
1819 H Street, K. W, , 
Washington, I), C. 20006 



March 10, 




1970 







K<y 



P 








D e a r D r , Ha yin.a n : 



The following is a brief feedback note concerning the Memphis 
training session. 

1 feel the greatest impact of the session was a positive affective 
shift toward Measurement in the Affective Domain. The Thurs day 
session and particularly the documentations provided by Dr.^ Rogers 
will contribute most to the behavioralization of this shift. i' ie “ 
sent at ions of actual affective evaluation examples were of more 
practical value than the historical and/or theoretical accounts of the 
measnreiTnents -which were given. 



The Research Council can certainly strengthen Id.p^il research 
and evaluation efforts- by maximizing interaction between the Groat 

City Schools /and also by continuing to update aha ctrengtheij &tafi 

conxpo ten cic & through training; se s s ions which .will. be c om & b e |te r 
a«d mb re spe c'if-i c with expo rie ncej; The Memphis; ac ss io n was ... 
certainly a step, in the right direction, 

SihCtirciy yours , 




GlbJln E. Queer / 

F iedd Rcso.arch 'Associate 



